Unsupervised Learning Project¶
Data Description:¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Domain:¶
Object recognition
Context:¶
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Attribute Information:¶
● All the features are geometric features extracted from the silhouette.
● All are numeric in nature.
Learning Outcomes:¶
● Exploratory Data Analysis
● Reduce number dimensions in the dataset with minimal information loss
● Train a model using Principle Components
Objective:¶
Apply dimensionality reduction technique – PCA and train a model using principle components instead of training the model using just the raw data.
Steps and tasks:¶
- Data pre-processing – Perform all the necessary preprocessing on the data ready to be fed to an Unsupervised algorithm (10 marks)
- Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why (10 points)
- Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn) (5 marks)
- Train a Support vector machine using the train set and get the accuracy on the test set (10 marks)
- Perform K-fold cross validation and get the cross validation score of the model (optional)
- Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data – (10 points)
- Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state) (20 marks)
- Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings (5 points)
References:¶
● Book on PCA ● Application of PCA for image compression
1. Data pre-processing – Perform all the necessary preprocessing on the data ready to be fed to an Unsupervised algorithm (10 marks)¶
In [307]:
# Importing the libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from sklearn.model_selection import train_test_split
In [308]:
# Reading the dataset
data = pd.read_csv('vehicle-1.csv')
data.sample(5)
Out[308]:
In [309]:
# Shape of the data is 846 rows and 19 columns
data.shape
Out[309]:
In [310]:
# There are some missing values in some of the rows.
data.info()
In [311]:
# Let's check the data distribution to decide how to impute the missing values
data.describe()
Out[311]:
- Most of the columns are whole numbers and the mean and median are almost the same, so we can impute the data using the median values.
In [312]:
# Lets treat the missing values and replace by the median of the column
from sklearn.impute import SimpleImputer
# Remove the target class variable before imputation and store it for future use
y = data['class']
X = data.drop(axis=1, columns=['class'])
cols = X.columns
impute = SimpleImputer(strategy='median')
X = pd.DataFrame(impute.fit_transform(X), columns=cols)
X.info()
- Now the data is clean without any missing values
2. Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why (10 points)¶
2.1 Finding out the Correlation between the independent variables¶
In [313]:
corr = X.corr()
corr
Out[313]:
In [314]:
# Let's check the heatmap to better visualize the correlation coefficients
plt.figure(figsize=(15,10))
sns.heatmap(corr, annot=True)
Out[314]:
2.2 Observations:¶
- From the heatmap, its very clear the columns circularity, scatter_ratio, elongatedness and pr.axis_rectangularity will be causing multicollinearity problem. Because these columns have multiple correlation coeeficient values greater than 0.9
- Remaining columns don't have as high correlation, so they need to be retained
- Pairplot indicates there are 3 clusters possible from the KDE plots which is the 3 types of vehicles
- The Pairplot also indicates there are lots of variables which are interdependent on each other which may impact the accuracy of the model
In [315]:
sns.pairplot(X, diag_kind='kde')
Out[315]:
2.3 To Drop columns or not?¶
To avoid multicolleniarity issue, one should drop the columns. However, in the later part of the project, we want to do Principal component analysis. So, its better to retain all original variables and capture the maximum variance possible in the PCA itself.
In [316]:
# Decided not to drop the columns.
# X.drop(columns=['circularity', 'scatter_ratio', 'elongatedness', 'pr.axis_rectangularity'], inplace=True)
# X.shape
3. Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn) (5 marks)¶
In [317]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=6)
X_train.head()
Out[317]:
4. Train a Support vector machine using the train set and get the accuracy on the test set (10 marks)¶
A) SVM with default parameters¶
In [318]:
from sklearn.svm import SVC
svc = SVC()
svc.fit(X_train, y_train)
print('Train set Accuracy: ', svc.score(X_train, y_train))
y_pred = svc.predict(X_test)
print('Test set Accuracy: ', svc.score(X_test, y_test))
In [319]:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
cm
Out[319]:
In [320]:
from sklearn.metrics import plot_confusion_matrix
plot_confusion_matrix(svc, X_test, y_test, values_format='g')
Out[320]:
5. Perform K-fold cross validation and get the cross validation score of the model (optional)¶
B) SVM with K-fold cross validation¶
In [321]:
from sklearn.model_selection import cross_validate, cross_val_score, cross_val_predict, GridSearchCV
param = {
'C' : [1.0, 2.0, 10.0],
'gamma' : ['scale', 'auto']
}
grid = GridSearchCV(svc, param, cv=5, refit=True)
grid.fit(X_train, y_train)
print('Train set Accuracy: ', grid.score(X_train, y_train))
print('Test set Accuracy: ', grid.score(X_test, y_test))
In [322]:
plot_confusion_matrix(grid, X_test, y_test, values_format='0.0f')
Out[322]:
6. Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data – (10 points)¶
6.1 Data Scaling¶
Need to scale the data before applying Principal Component Analysis
In [323]:
from scipy.stats import zscore
X_scaled = X.apply(zscore)
X_scaled.head()
Out[323]:
In [324]:
X_scaled.shape
Out[324]:
6.2 Principal Component Analysis¶
In [325]:
from sklearn.decomposition import PCA
pca = PCA(n_components=14, random_state=10)
pca.fit(X_scaled)
Out[325]:
In [326]:
# Checking the explained variance by each Principal component,
# it looks like the first 4 components explain the majority of the variance in the data
pca.explained_variance_
Out[326]:
In [327]:
# Checking the Explained variance ratio
pca.explained_variance_ratio_
Out[327]:
In [328]:
# Plotting a bar graph to visualize the comparitive explained variance
plt.figure(figsize=(10,6))
plt.xlabel('Principal Components')
plt.ylabel('Explained variance Ratio')
plt.bar(list(range(1,15)),pca.explained_variance_ratio_);
In [329]:
# Plotting a step graph to view the cummulative explained variance
plt.figure(figsize=(10,6))
plt.xlabel('Principal Components')
plt.ylabel('Cummulative Explained variance')
plt.step(list(range(1,15)),np.cumsum(pca.explained_variance_ratio_),where='mid');
In [330]:
np.cumsum(pca.explained_variance_ratio_)
Out[330]:
6.3 Dimensionality Reduction¶
From the above Step graph and the above cummulative explained variance numbers, it appears that 7 Principal components are required to explain more than 95% of the total variance in the dataset
In [331]:
pca7 = PCA(n_components=7)
pca7.fit(X_scaled)
Xpca7 = pd.DataFrame(pca7.transform(X_scaled))
Xpca7.head()
Out[331]:
In [332]:
sns.pairplot(Xpca7);
Result: As a result of PCA, the 14 attributes have been extracted into 7 dimensions which are independent of each other¶
7. Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state) (20 marks)¶
In [333]:
# Generating the new Train Test set using the same random state.
X_train, X_test, y_train, y_test = train_test_split(Xpca7, y, test_size=0.3, random_state=6)
X_train.head()
Out[333]:
C) SVM with Principal components¶
In [334]:
from sklearn.svm import SVC
svc = SVC()
svc.fit(X_train, y_train)
print('Train set Accuracy: ', svc.score(X_train, y_train))
y_pred = svc.predict(X_test)
print('Test set Accuracy: ', svc.score(X_test, y_test))
In [335]:
plot_confusion_matrix(svc, X_test, y_test, values_format='0.0f')
Out[335]:
D) SVM with Principal components using K-fold cross validation¶
In [336]:
from sklearn.model_selection import cross_validate, cross_val_score, cross_val_predict, GridSearchCV
param = {
'C' : [1.0, 2.0, 10.0],
'gamma' : ['scale', 'auto']
}
grid = GridSearchCV(svc, param, cv=5, refit=True)
grid.fit(X_train, y_train)
print('Train set Accuracy: ', grid.score(X_train, y_train))
print('Test set Accuracy: ', grid.score(X_test, y_test))
In [337]:
plot_confusion_matrix(grid, X_test, y_test, values_format='0.0f')
Out[337]:
8. Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings (5 points)¶
Conclusion:¶
- Comparing the test dataset accuracy scores:
- Default SVM: 69.29 %
- SVM with 5-fold cross validation: 87.01 %
- SVM after PCA: 91.73 %
- SVM after PCA with 5-fold cross validation: 92.12 %
- Findings:
- PCA improves the model accuracy by a considerable amount ~ almost 20% in this case
- PCA reduces the dimensionality of the dataset from 14 to 7 attributes, thus saving a lot of computational power
- The amount of variance captured by PCA is 95% less than using the complete dataset, but the model accuracy is still higher than that of the best SVM parameters determined by K-fold cross validation technique
In [ ]: